The present invention relates generally to a highly available directory service, and in particular to apparatus and methods for a highly available directory service in Distributed Computing Environment.
The Distributed Computing Environment (DCE) is a suite of software utilities and operating system extensions that can be used to create applications on networks of heterogeneous hardware--PCs, Unix workstations, minicomputers and mainframes. The DCE is a standard developed by the Open Software Foundation (OSF) and is designed to simplify building heterogeneous client/server applications. Several services are provided by DCE: Remote Procedure Call (RPC) facilitates client-server communication, so that an application can effectively access resources distributed across a network; Security Service authenticates the identities of users, authorizes access to resources on a distributed network, and provides user and server account management; Directory Service provides a single naming model throughout the distributed environment; Time Service synchronizes the system clocks throughout the network; Threads Service provides multiple threads of execution capability; and, Distributed File Service provides access to files across a network.
Directory Service performs typical naming services in a distributed computing environment and acts as a central repository for information about resources in the distributed system. Typical resources are users, machines, and RPC-based services. The information consists of the name of the resource and its associated attributes. Typical attributes include a user's home directory or the location of an RPC-based server.
The Directory Service provides a distributed and replicated repository for information on various resources of a distributed system, such as location of servers, and the services they offer. Clients use the name space to locate service providers in a well-defined, network-independent manner. The Directory Service consists of a Global Directory Service (GDS) that spans different administrative domains and a Cell Directory Service (CDS). A cell is the unit of management and administration in DCE and typically consists of a group of machines on a local area network. CDS maintains a local, hierarchical name space within the cell. The Cell Directory Service manages a database of information about the resources in a group of machines called a DCE cell. The GDS enables intercell communications by locating cells which have been registered in the global naming environment. The present invention focuses on enhancing the availability and accuracy of CDS.
The CDS name space is partitioned over one or more CDS servers to improve the scalability and availability of the service. A CDS server maintains an in-memory database, called a clearinghouse, to store the cell's name space or portions thereof. Clients access a server using the DCE RPC mechanism. CDS servers typically announce their presence by broadcasting their network addresses periodically over the local area network.
The performance and availability of CDS is crucial to applications built using DCE, especially for those applications having high availability requirements. To address these requirements, CDS names may be replicated on different servers. The replication model follows the primary copy approach. The unit of replication is a directory, but subdirectories of a replicated directory are not automatically replicated. One physical copy of a directory is designated a master replica, while the others are designated read-only replicas. Updates can only be made to the master replica. Lookups can be handled by any server that contains a copy of the directory. This design helps offload lookup requests from the site containing the master copy of an entry. Should the server containing the master copy of a directory entry fail, CDS continues to service lookup requests for that directory through the available replicas, although no update requests can be handled until the server containing the master replica recovers from failure or a new master replica is created.
DCE allows flexibility in specifying which directory entries are to be replicated and the degree of consistency across replicas. The CDS propagates updates to the read-only replicas, either immediately after performing the update on the master copy or after a certain amount of time, depending on the desired degree of consistency. The propagation occurs on a best-effort basis. If a failure occurs in communication, the propagation is periodically retried in the background until it succeeds.
There are several deficiencies in CDS specifications that affect naming service availability and correctness. The DCE CDS falls short in providing the necessary degree of availability. Furthermore, CDS often returns inconsistent or incoherent information as a result of a lookup, making correct operation impossible.
CDS propagates an update to read-only replicas after it sends the corresponding reply to the client. Consider the scenario where a DCE application sends a request to advertise its services in the name space. The server containing the master replica of the corresponding name space entry may fail after sending the reply to the application but before propagating the updates to the read-only replicas. In such a case the application's advertisement will not be available until the server maintaining the master replica recovers, which may take a long time. The application itself is not aware of the problem since it has received a reply indicating that the advertisement was properly handled and therefore would not attempt any corrective action.
Another problem related to CDS operation is that lookups performed on CDS replicas do not necessarily return correct information. This occurs if the master replica does not immediately propagate updates to the other replicas, or if a communication failure prevents such updates from reaching the replicas. Applications, therefore, cannot trust information returned by read-only replicas.
In the case of failure of a CDS server that maintained the master replica of a directory, CDS allows a read-only replica of the directory, on another server, to be promoted to a master replica. There are three problems with using this mechanism. First, this mechanism fails if the security server was also running on the same machine as the failed CDS server (as is common in commercial installations). As a result, the system is effectively susceptible to a single point of failure with respect to update requests for some directory entries. Second, this mechanism needs to be executed for every directory for which the failed server maintained a master replica. Obtaining a list of each such directory is cumbersome and time consuming. Finally, the entire reconfiguration mechanism requires manual intervention and can take many minutes to execute, rendering it inadequate when a high degree of availability is required.
Another related problem is that CDS does not support any form of automatic failure detection or reconfiguration. The failure of a CDS server is detected by RPC timeouts in clients invoking operations on the server. Failed servers must then be manually restarted and configured into the system, as described earlier.
What is needed then is a system which overcomes the above problems in order to provide a robust and accurate directory service.